LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation

Abstract Standard multi-task benchmarks are essential for developing pretraining models that can generalize to various downstream tasks. Existing natural language processing (NLP) usually focus only on understanding or generating short texts. However, long text modeling requires many distinct abilities in contrast texts, such as the of long-range discourse and commonsense relations, coherence controllability generation. The lack standardized makes it difficult assess these a model fairly compare different models, especially Chinese models. Therefore, we propose story-centric benchmark named LOT evaluating modeling, which aggregates two tasks generation We construct new datasets based human-written stories with hundreds words. Furthermore, release an encoder-decoder-based LongLM up 1 billion parameters. pretrain 120G novels generative including infilling conditional continuation. Extensive experiments show outperforms similar-sized substantially both LOT.